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Abstract 

This  study  reports  some  preliminary  results  of  a  research  agenda  that  seeks  to 
address  the  absence  of  tested  metrics  to  provide  early  indication  of  the  acquisition  risks  of 
interdependent  programs.  The  overall  goal  of  this  research  is  to  forge  new  ground  on 
uncovering  early  indicators  of  interdependency  acquisition  risk,  so  appropriate  governance 
mechanisms  can  then  be  isolated.  Funding  will  allow  the  ability  to  1)  expand  on  an  existing 
database  of  Major  Defense  Acquisition  Program  (MDAPs)  performance  data,  2)  analyze  the 
MDAPs  to  characterize  the  risks  attributable  to  interdependence,  and  3)  determine  whether 
acquisition  setbacks  cascade  to  downstream  interdependent  MDAP  programs.  The 
deliverables  of  the  effort  are  1 )  a  code  book  of  DoD  data  acquisition  items  that  can  be 
employed  in  future  research  efforts,  and  2)  the  results  of  the  investigation  of  the  cascading 
risks  of  interdependent  acquisition  efforts. 


In  short,  preliminary  results  indicate  that  perceptions  of  risk  may  prove  influential  on 
downstream  program  performance.  In  terms  of  the  direct  influence  that  an  upstream 
program’s  performance  might  exert  on  downstream  program  performance,  weak,  but 
statistically  significant,  relationships  were  noted  in  three  areas.  The  next  steps  of  the 
research  are  to  1)  expand  the  dataset  to  include  FY  2009  data,  2)  document  acquisition 
data,  3)  collect  a  number  of  indicators  on  program  interdependency,  and  4)  test  a  number  of 
interdependency  diversity  metrics  in  terms  of  their  ability  to  provide  insights  on  program 
performance. 
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Introduction 


This  research  effort  addresses  two  critical  problems:  1)  Data  on  major  defense 
acquisition  programs  is  piecemeal  and  fragmented,  thus  hindering  acquisition  research;  2) 
There  is  an  absence  of  tested  metrics  to  provide  early  indication  of  the  acquisition  risks  of 
interdependent  programs. 

A  wealth  of  research  indicates  that  the  use  of  programmatic  networks  in  the  public 
sector  is  clearly  on  the  rise  (Weber  &  Khademian,  2008).  Noting  that  firms  do  not  act  by 
themselves,  Granovetter  (1973)  witnessed  that  organizations  are  deeply  embedded  in 
“networks  of  external  relationships”  that  influence  the  exchange  of  resources  and 
capabilities  among  them.  Increasingly,  state-,  local-,  and  federal-level  agencies  are  turning 
to  joint  interdependent  programs  to  address  gaps  that  only  cross  collaborative  initiatives  can 
span.  Yet,  as  discussed  below,  the  study  of  interdependency  and  its  effects  on  program 
performance  have  yielded  too  few  tangible  results.  For  purposes  of  the  discussion  below, 
jointness,  interdependency,  exchange,  and  partnerships,  all  refer  to  a  similar  concept:  the 
notion  that  autonomous  organizations  build  relationships  to  provide  capabilities  that,  when 
looked  at  in  totality,  form  network  structures.  Additionally,  at  the  individual  pair-wise  level, 
the  exchanges  are  manifested  as  explicit  transactions. 

Scholars  have  long  contended  that  many  contemporary  policy  challenges,  and  their 
associated  solutions,  lie  across  organizational  domains  outside  the  jurisdiction  of  any  one 
agency  (Gage  &  Mandell,  1990;  Alexander,  1995;  Agranoff,  2003).  Milward  and  Provan 
(2001)  show  that  public  policy  arenas  are  inherently  crosscutting;  the  requisite  knowledge 
and  corresponding  solutions  are  not  localized,  but  are  instead  distributed  across  a  range  of 
agencies  and  organizations.  The  DoD’s  transformation  to  joint  capabilities  is  in  keeping  with 
the  ongoing  trends.  Historically,  acquisition  investments  at  the  DoD  had  been  proposed  as 
individual  materiel  solutions,  typically  championed  by  the  armed  service  for  which  the 
product  was  to  be  obtained.  This  gave  rise  to  discrete  systems  designed  in  accordance  with 
the  individual  service  requirements.  When  called  upon  to  operate  in  a  joint,  multi-service 
environment,  these  systems  exhibited  problems  interacting  effectively  with  other  service 
systems. 

The  Transformation  to  Joint  Capabilities  attempts  to  provide  military  forces  with  the 
capability  to  adapt  quickly  to  new  challenges  and  unexpected  circumstances  by  leveraging  a 
wider  range  of  assets.  Central  to  the  Transformation  was  the  desire  for  enhanced 
coordination  among  agencies  and  across  all  levels  of  government  (coalition,  federal,  state, 
and  local).  In  addressing  the  need  for  interagency  cooperation,  Vice  Chairman  of  the  Joint 
Chiefs  of  Staff  Admiral  Giambastiani  (2004)  claimed  that  the  integrated  force  had  to  become 
interdependent.  That  is,  it  must  be  capabilities-based,  collaborative,  and  network  centric. 
Military  efforts  require  the  ability  to  conduct  high-level,  or  large-scale,  vertical  and  horizontal 
collaboration.  That  means  up  and  down  the  chain  of  command  and  across  all  capabilities 
and  forces. 

While  DoD  agencies  are  expected  to  embrace  joint  capabilities,  literature  findings 
regarding  the  risks  and  best  practice  mechanisms  of  joint  interdependent  activities  lag  far 
behind.  Whereas  early  research  did  provide  some  insights,  the  research  activities  have 
stalled  and  progress  is  lacking.  For  example,  back  in  1937,  Coase  found  that 
interdependencies  are  based  on  mutual  exchanges  that  can  be  examined  at  the  transaction 
level.  He  argued  that  these  transactions  accrued  costs  that  could  be  attributed  to 
establishing  the  rules  of  engagement,  enforcing  agreements,  and  monitoring  compliance. 
Unfortunately,  specific  cost  functions  were  never  isolated. 
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In  1967,  Thompson  contributed  to  the  research  by  offering  a  tripartite  model  that 
focused  on  the  configurations  of  the  transaction  flows.  Sequential  flows  involved  handoffs 
between  partners.  Pooled  flows  involved  partners  that  drew  down  from  a  common  source  of 
assets  and  the  flows  of  reciprocal  relationships  involved  feedback  mechanisms.  Much  of 
the  research  to  date  has  been  based  on  the  anecdotal  findings  of  small  n  case  studies  (Isett 
&  Provan,  2005;  Meier  &  O’Toole,  2008).  While  the  three  configurations  provide  a  starting 
point  for  understanding  interdependent  activities,  the  reality  of  today’s  activities  are  far  more 
complex.  In  short,  it  is  not  unusual  for  the  acquisition  or  production  of  a  service  to 
incorporate  multiple  configurations  with  resources  flowing  in  and  out  across  organizations  of 
public  and  private  entities.  As  such,  the  most  common  configuration  is  the  “mixed”  pattern 
incorporating  all  three  of  the  configurations  and  a  wide  array  of  nodes,  assets,  channels  and 
zones. 


Exchange  theorists  argue  that  organizations  develop  interdependent  relationships 
with  other  organizational  entities  to  either  obtain  critical  resources  or  provide  critical 
capabilities.  They  also  assert  that  interdependent  relationships  exhibit  high  levels  of 
uncertainty  due  to  participant 
constraints  (Miles  &  Snow,  1978). 

Shirking  or  defection  of  a  network 
member  can  have  dire  consequences 
on  the  survival  and  performance  of 
the  network  in  total  and  network 
participants  in  general.  Because  of 
the  nature  and  influence  of  the  ties  or 
interdependencies  that  bind 
organizations,  Levinthal’s  (1997) 
research  indicated  that  increasing  the 
density  of  the  interdependencies  that 
connect  the  organizations  affects  the 
complexity  of  the  “landscape”  in  which 
it  operates.  Levinthal  (1997)  finds  that 
these  interconnections  or  flows  can 
yield  nonlinear  consequences  that 
often  involve  multiplier  effects  based 
on  the  nature  of  the 
interdependencies  in  the  system. 

Apparently,  the  value  chain  of  the  joint 
capabilities  is  laden  with  junctions  and  bifurcations  where  delay,  defection,  or  shirking  can 
occur.  In  fact,  in  1999,  Rosen  argued  that  the  uncertainty  that  arises  from  a  relationship  is 
the  definition  of  “complexity.”  And  that  “complexity”  can  only  be  understood  by  examining 
the  links  that  bind.  If  Rosen  is  correct,  then  DoD  acquisition  is  reaching  unprecedented 
complexity.  A  network  analysis  of  MDAP  data  interdependencies  suggests  overwhelming 
complexity  (see  Figure  1).  The  78  MDAPs  in  2005  demonstrated  989  unique  links  to  other 
MDAPs  as  well  as  non-MDAP  programs.  The  yellow  links  indicate  medium  risk  relationships 
and  the  red  links  show  high  risk  links. 

Despite  the  activities,  ten  years  ago  Agranoff  and  McGuire  (2001 )  wrote  that  “there 
are  many  more  questions  than  answers  in  network  management”  and  the  assertion 
continues  to  ring  true.  Apparently,  the  field  is  rich  in  anecdotal  findings  but  poor  in  empirical 
evidence  (Alexander,  1995).  Oliver  and  Ebers  (1998)  liken  the  state  of  the  field  as  a  messy 
situation  marked  by  a  cacophony  of  heterogeneous  concepts,  theories,  and  research 
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Figure  1:  MDAP  Data  Interdependencies  in  2005 


results.  While  the  growth  is  clearly  on  the  rise,  DoD  acquisition  is  moving  forward  with  little 
insights  into  the  risks  and  threats  of  joint  efforts.  Without  a  deep  understanding  of  the  risks 
and  threats  that  interdependent  efforts  encounter,  governance  mechanisms  that  can  help  to 
insure  acquisition  success  are  beyond  reach.  Given  the  pace  at  which  joint  efforts  are 
pursued,  early  indicators  of  acquisition  risk  are  needed  to  help  isolate  the  critical 
governance  mechanisms  that  will  mitigate  performance  shortfalls. 

Additionally,  of  utmost  concern  is  the  state  of  current  acquisition  data.  While  several 
initiatives  are  underway  to  compile  the  sundry  datasets  the  fragmented  acquisition  data  puts 
decision-making  at  risk  for  Type  I  and  Type  II  errors  (false  positive  and  false  negative, 
respectively).  Funding  for  this  research  effort  will  allow  the  ability  to  integrate,  cleanse,  and 
normalize  authoritative  datasets  for  the  purpose  of  advanced  research.  It  will  also  provide 
the  ability  to  document  the  acquisition  data  for  future  research  purposes.  The 
documentation  will  allow  the  research  community  to  forge  new  insights  in  the  acquisition 
arena. 


The  study  of  DoD  acquisition  efforts  for  gaining  insights  on  interdependency  is 
especially  fruitful.  First,  the  DoD  has  rich,  but  fragmented  and  piecemeal,  datasets  on  some 
of  the  important  key  interdependencies  of  major  defense  acquisition  programs.  Second,  the 
movement  toward  joint  capabilities  makes  an  understanding  of  interdependencies  especially 
critical.  Finally,  given  the  frequency  with  which  government  agencies  are  moving  toward 
joint  initiatives,  findings  based  on  DoD  programs  may  prove  instrumental  to  a  wide  range 
audience.  The  research  below  examines  the  ties  that  bind  organizations  in  light  of  two 
different  types  of  transactions:  data  ties  and  funding  ties.  And  it  begs  the  questions:  do  the 
ties  result  in  cascading  acquisition  risks?  The  research  will  act  as  a  catalyst  for  a  long-term 
research  program  on  the  risks  and  governance  mechanisms  of  joint  acquisition  initiatives. 

Research  Objectives:  The  objectives  of  the  research  are  as  follows: 

1 .  To  map  program  interdependence  to  reveal  the  directionality  of  influence  (i.e., 
“upstreamTdownstream”)  of  cause-effect  relationships, 

2.  To  test  the  cascading  risks  that  upstream  programs  exert  on  downstream 
programs  in  light  of  data  and  funding  exchanges, 

3.  To  test  the  extent  to  which  the  cost  overruns  of  upstream  programs  cascade 
on  to  interdependent  downstream  programs, 

4.  To  test  the  extent  to  which  schedule  delays  cascade  on  to  interdependent 
downstream  programs, 

5.  To  use  the  findings  to  make  recommendations  on  potential  governance 
mechanisms  that  may  prove  capable  of  mitigating  the  risks  of 
interdependencies,  and 

6.  To  provide  a  research  code  book  of  acquisition  data  elements  for  future 
research  efforts. 

The  remaining  discussion  provides  an  interim  report  on  some  of  the  findings  to  date, 
based  on  the  interdependent  activities  identified  in  the  Defense  Acquisition  Executive 
Reviews  (DAES).  The  findings  below  examine  the  influence  of  interdependencies  on  a 
number  of  program  performance  measures. 
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Research  Methods:  Because  of  data  availability  issues,  the  unit  of  analysis  for 
this  research  was  restricted  to  Major  Defense  Acquisition  Programs  (MDAPs).  Starting  in  the 
2005  time  period,  Major  Defense  Acquisition  Program  managers  were  asked  to  provide 
reports  on  what  they  considered  to  be  the  critical  interdependencies  of  their  given  program. 
They  identified  upstream  and  downstream  connections  and  indicated  the  perceived  risk 
level  (red,  yellow,  green).  Because  of  a  small  number  of  red  risks,  the  risk  variable  was 
recoded  to  reflect  “no-risk,”  “risky.”  Hence  the  “risk”  variable  is  binomial  in  nature.  The 
research  findings  described  below  are  based  on  the  influences  these  interdependencies 
might  exert  on  program  performance. 

Program  performance  is  considered  from  multiple  vantage  points.  Table  1  provides 
all  the  variables  used  in  this  research  and  the  documents  they  were  derived  from.  In  short, 
performance  is  based  on  1 )  annual  cost  variance  figures  (for  total  cost  variance  and  the 
subsets  of  schedule,  estimation,  and  engineering  cost  variance  in  millions),  2)  DAES 
breaches  (schedule,  performance,  and  Research,  Development,  Testing,  and  Evaluation 
(RDT&E)  breaches),  and  3)  percent  cost  growth  from  the  original  RDT&E  estimates.  All 
variables  were  derived  from  Selected  Annual  Reports  (SARs)  and  DAES  reports.  Because 
2005  marked  the  first  year  that  the  DAES  reports  began  reporting  interdependencies,  the 
analysis  reported  below  is  based  on  MDAP  performance  in  fiscal  years  2005,  2006,  and 
2007  (note:  SAR  reports  were  not  reported  in  FY2008  due  to  the  new  presidential 
administration). 


Table  1.  Variables  Used  in  the  Research 


Unit  of  Analysis:  Major  Defense  Acquisition  Programs 

Variable 

Data  Source 

Count  of  APB  Schedule  Breaches 

Count  of  APB  Performance  Breaches 

Count  of  APB  RDT&E  Breaches 

Count  of  APB  PAUC  Breaches 

DAES 

Total  Cost  Variance 

Engineering  Cost  Variance 

Schedule  Cost  Variance 

Estimation  Cost  Variance 

SAR 

Percent  Cost  Growth 

SAR 

Data  Exchange  Interdependencies 

DAES  Interdependency  Charts 

As  identified  above,  MDAP  program  managers  were  asked  to  provide  insight  on  what 
they  perceived  to  be  the  program’s  interdependencies.  They  also  reported  on  the  direction 
of  the  interdependency  (inbound,  outbound,  and  bidirectional)  and  the  risk  of  the 
interdependency.  The  risk  is  based  on  the  sender’s  perceived  risk  with  a  downstream 
receiver.  Because  performance  data  on  non-MDAP  programs  were  not  available  for 
analysis,  the  findings  considered  only  the  interdependencies  that  existed  among  MDAPs. 

Findings:  Two  major  sets  of  findings  are  discussed  below.  The  first  is  based  on  the 
influence  of  the  sender’s  perceived  risk  with  the  downstream  receiver’s  performance  (during 
years  2005-2007).  The  sender’s  “perception  of  risk”  is  also  considered  in  light  of  their  own 
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individual  performance  (during  years  2005-2007).  To  measure  the  sender’s  perceived  risk 
on  their  individual  performance,  the  mean  of  the  risk  of  all  the  relationships  was  calculated 
to  provide  an  overall  “risk”  level  for  each  MDAP.  The  second  set  of  findings  is  based  on  the 
sender’s  performance  and  its  influence  on  the  receiver’s  performance.  Table  2  provides  the 
means  and  standard  deviations  of  all  variables  used  in  the  research.  The  number  of 
downstream  programs  per  upstream  MDAP  ranged  from  1  to  23,  with  a  mean  of  5.  A  total 
of  873  relationships  were  analyzed  over  the  three-year  time  period. 


Table  2.  Descriptive  Statistics  of  Variables  Employed  in  the  Research 


N 

Mean 

Std. 

Deviation 

Upstream  APB  Schedule  Breaches 

269 

.36 

.481 

Upstream  APB  RDT&E  Breaches 

522 

.09 

.289 

Upstream  APB  PAUC  Breaches 

522 

.08 

.266 

Upstream  APB  Performance  Breaches 

522 

.13 

.333 

Downstream  APB  Schedule  Breaches 

518 

.27 

.444 

Downstream  APB  Performance  Breaches 

522 

.06 

.240 

Downstream  APB  RDT&E  Breaches 

522 

.11 

.319 

Downstream  APB  PAUC  Breaches 

522 

.11 

.312 

Upstream  Perceived  Risk 

522 

1.1073 

.30977 

N 

Mean 

Std. 

Deviation 

Upstream  Percent  Cost  Growth 

840 

.08 

1.00 

Upstream  Engineering  Cost  Variance 

711 

5.53 

91.06 

Upstream  Schedule  Cost  Variance 

711 

4.76 

38.31 

Upstream  Estimation  Cost  Variance 

711 

.32 

159.33 

Upstream  Total  Cost  Variance 

840 

4.75 

208.20 

Downstream  Percent  Cost  Growth 

446 

.014 

.10 

Downstream  Engineering  Cost  Variance 

351 

-6.12 

219.25 

Downstream  Schedule  Cost  Variance 

351 

3.77 

25.05 

Downstream  Estimation  Cost  Variance 

351 

-7.95 

324.36 

Downstream  Total  Cost  Variance 

449 

-38.79 

400.26 

Interdependency  Risk.  The  first  set  of  tests  sought  to  determine  if  the  sender’s 
perceived  risk  of  the  relationship  influenced  its  partnering  receiver.  In  terms  of  the  “partner 
risk”  variable,  xx%  of  the  MDAP  program  managers  identified  no  risk  in  the  partnerships. 
Xx%  indicated  some  degree  of  risk  in  the  relationships.  Of  the  56  programs  that  indicated 
some  risk,  the  risk  ranged  from  a  low  of  1 .1  to  a  high  of  2  (recall  that  the  variable  ranges 
from  “1”  to  “2,”  with  “1”  indicating  no  risk).  Correlation  coefficients  were  then  obtained  (see 
Table  3).  The  data  show  that  the  Manager’s  Perception  of  Risk  is  negatively  with  the 
partner’s  total  cost  variance,  engineering  cost  variance,  and  estimation  cost  variance. 
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Interestingly,  risk  was  correlated  with  the  downstream  partner’s  performance,  RDT&E,  and 
PAUC  breaches,  but  in  a  positive  direction.  The  fact  that  an  upstream  partner’s  perception 
of  risk  might  result  in  an  increase  in  the  number  of  DAES’  breaches  illustrates  the 
detrimental  influence  that  upstream  programs  might  exert  on  their  downstream  partners. 
But,  why  the  negative  relationship  with  cost  variance?  Why  would  an  increasing  perception 
of  risk  on  the  upstream  program  result  in  reducing  the  cost  variance  of  its  downstream 
partners?  Perhaps  the  answer  lies  in  the  old  adage  that  perceptions  of  risk  result  in 
increased  attention.  Perhaps,  under  high-risk  situations,  program  managers  are  more 
cognizant  of  the  risk  and  act  to  mitigate  the  detrimental  effects.  More  research  is  clearly 
warranted  to  tease  out  why  these  correlations  are  demonstrated. 

Table  3.  Bivariate  Correlation  with  MDAP  Program  Manager’s  Perception  of  Risk 


MDAP’s  Percent  Cost  Growth 

-.024 

MDAP’s  Total  Cost  Variance 

.019 

MDAP’s  Engineering  Cost  Variance 

-.081* 

MDAP’s  Schedule  Cost  Variance 

-.007 

MDAP’s  Estimation  Cost  Variance 

.05 

MDAP’s  Schedule  Breaches 

.05 

MDAP’s  Performance  Breaches 

.11* 

MDAP’s  RDT&E  Breaches 

.13** 

MDAP’s  PAUC  Breaches 

-.05 

Downstream  MDAP’s  Percent  Cost  Growth 

-.07 

Downstream  MDAP’s  Total  Cost  Variance 

/j  2** 

Downstream  MDAP’s  Engineering  Cost  Variance 

-.22** 

Downstream  MDAP’s  Schedule  Cost  Variance 

-.05 

Downstream  MDAP’s  Estimation  Cost  Variance 

-.11* 

Downstream  MDAP’s  Schedule  Breaches 

.07 

Downstream  MDAP’s  Performance  Breaches 

.13** 

Downstream  MDAP’s  RDT&E  Breaches 

.09* 

Downstream  MDAP’s  PAUC  Breaches 

.12** 

*  p  <  .05  **  p  <  .00 

The  next  step  of  the  “risk”  analysis  sought  to  isolate  whether  the  MDAP’s  mean  risk 
score  influenced  their  own  specific  performance.  The  results  show  that  the  manager’s 
perception  of  risk  is  positively  correlated  with  the  program’s  engineering  cost  variance. 
Outside  of  engineering  cost  variance,  despite  recognition  of  risk,  no  notable  correlations 
were  demonstrated  in  terms  of  DAES  breaches,  cost  variance,  or  cost  growth. 

The  second  set  of  tests  sought  to  identify  whether  the  performance  of  upstream 
programs  exert  influence  on  their  downstream  partner’s  performance.  Because  an 
upstream  program’s  influence  would  not  expect  to  have  an  immediate  effect,  the  data  were 
lagged  a  year.  In  other  words,  one  might  expect  that  the  negative  effects  of  an  upstream 
program  would  influence  their  downstream  partners  one  year  out.  Table  4  shows  the  results 
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of  the  sender’s  performance  on  the  downstream  partners.  The  results  show  little  influence 
on  APB  breaches.  A  weak,  but  statistically  significant,  relationship  was  demonstrated 
between  the  number  of  upstream  program  RDT&E  breaches  and  the  number  of  downstream 
program  RDT&E  breaches.  In  terms  of  the  influence  of  the  upstream  program’s  percent 
growth,  it  showed  no  correlation  with  the  downstream  program’s  percent  growth.  Two  of  the 
cost  variance  relationships  also  showed  weak,  but  statistically  significant,  correlations.  The 
sender’s  total  cost  variance  appeared  to  exert  some  influence  on  the  downstream  program’s 
schedule  variance.  In  addition,  as  the  sender’s  engineering  cost  variance  rose,  a 
subsequent  rise  was  also  noted  in  the  downstream  program’s  percent  cost  growth. 

Table  4.  The  Influence  of  the  Sender’s  Performance  on  Downstream  Programs 

FY  2005-2007  (Lag  of  One  Year) 


SENDER  MDAP 

APB 

Schedule 

Breaches 

APB 

RDT&E 

Breaches 

APB 

Performance 

Breaches 

APB 

PAUC 

Breaches 

Total 

Cost 

Variance 

Engineering 

Cost 

Variance 

Schedule 

Cost 

Variance 

Estimatio 
n  Cost 
Variance 

Percent 

Cost 

Growth 

D 

O 

APB 

Schedule 

-.03 

.04 

-.01 

.05 

W 

APB  RDT&E 

-.00 

.07* 

.02 

-.05 

N 

S 

T 

APB 

Performance 

.06 

.04 

-.00 

.01 

APB  PAUC 

-.03 

.05 

-.02 

.00 

1 

R 

Total  Cost 
Variance 

-.02 

.09 

-.06 

.01 

-.01 

E 

A 

M 

Engineering 

Cost 

Variance 

.02 

.09 

-.06 

.02 

-.01 

M 

Schedule 

Cost 

Variance 

.09* 

-.04 

.05 

-.02 

-.00 

D 

A 

p 

Estimation 

Cost 

Variance 

-.08 

.09 

.02 

.03 

.02 

Percent 

Cost  Growth 

-.02 

.12** 

-.00 

.02 

-.00 

Conclusions 

The  results  discussed  above  are  the  preliminary  interim  results  of  a  segment  of 
research  that  seeks  to  identify  the  influences  that  interdependencies  might  exert  on 
acquisition  program  performance.  In  short,  the  results  indicate  that  perceptions  of  risk  may 
prove  influential  on  downstream  program  performance.  In  terms  of  the  direct  influence  that 
an  upstream  program’s  performance  might  exert  on  downstream  program  performance, 
weak,  but  statistically  significant,  relationships  were  noted  in  three  areas.  In  subsequent 
months,  we  will  complete  the  data  collection  effort  and  construct  and  test  a  series  of 
interdependency  metrics  on  program  performance.  The  data  will  be  modeled  using 
traditional  statistical  approaches  to  assess  causality.  Additionally,  we  will  employ  Markov 
Decision  Process  (MDP)-based  methods  (Puterman,  1994)  to  take  into  account  the  cost  and 
schedule  variance  specifications  from  the  n-ordered  downstream  programs  and  produce  a 
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specification  of  the  best  possible  budget  trimming  options  for  the  decision-maker.  Formally, 
a  MDP  is  a  probabilistic  model  of  a  sequential  decision  problem,  in  which  states  can  be 
perceived  exactly,  and  the  current  state  and  action  selected  determine  a  probability 
distribution  on  future  states  (Bertsekas,  1987).  Specifically,  the  outcome  of  applying  an 
action  to  a  state  depends  only  on  the  current  action  and  state  (and  not  on  preceding  actions 
or  states).  We  assume  that  state  changes  in  our  model  occur  only  at  discrete  instances  of 
time,  allowing  us  to  model  the  network  as  a  discrete  event  dynamic  system  (DEDS)  and  plan 
to  employ  MDPs.  Our  model  facilitates  the  data  acquisition  process  since  we  iteratively 
refine  the  state  features  critical  to  the  decision-making.  The  action  space  will  capture 
information  about  the  funders,  including  changes  in  level  of  funding.  We  plan  to  model  the 
probability  of  transitions  from  one  state  to  another  empirically  by  using  existing  data.  The 
“Reward”  function  will  be  the  presence  of  a  schedule  delay  and  cost  variance  that  occurs  in 
n-ordered  downstream  programs.  Hence,  we  will  be  assessing  various  interdependency 
metrics  in  light  of  statistical  and  MDP  methods  to  isolate  the  most  feasible  method  for 
understanding  interdependencies. 

In  short,  the  next  steps  of  the  research  are  to  1 )  expand  the  dataset  to  include  FY 
2009  data,  2)  document  existing  acquisition  data  sources,  3)  collect  a  number  of  indicators 
on  program  interdependency,  and  4)  test  a  number  of  interdependency  diversity  metrics  in 
terms  of  their  ability  to  provide  insights  on  program  performance. 
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2003  -2010  Sponsored  Research  Topics 


Acquisition  Management 


Acquiring  Combat  Capability  via  Public-Private  Partnerships  (PPPs) 

BCA:  Contractor  vs.  Organic  Growth 
Defense  Industry  Consolidation 
EU-US  Defense  Industrial  Relationships 

Knowledge  Value  Added  (KVA)  +  Real  Options  (RO)  Applied  to  Shipyard 
Planning  Processes 

Managing  the  Services  Supply  Chain 

MOSA  Contracting  Implications 

Portfolio  Optimization  via  KVA  +  RO 

Private  Military  Sector 

Software  Requirements  for  OA 

Spiral  Development 

Strategy  for  Defense  Acquisition  Research 

The  Software,  Hardware  Asset  Reuse  Enterprise  (SHARE)  repository 


Contract  Management 


Commodity  Sourcing  Strategies 
Contracting  Government  Procurement  Functions 
Contractors  in  21st-century  Combat  Zone 
Joint  Contingency  Contracting 

Model  for  Optimizing  Contingency  Contracting,  Planning  and  Execution 

Navy  Contract  Writing  Guide 

Past  Performance  in  Source  Selection 

Strategic  Contingency  Contracting 

Transforming  DoD  Contract  Closeout 

USAF  Energy  Savings  Performance  Contracts 

USAF  IT  Commodity  Council 

USMC  Contingency  Contracting 


Financial 


Management 


Acquisitions  via  Leasing:  MPS  case 


Budget  Scoring 


Budgeting  for  Capabilities-based  Planning 
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Capital  Budgeting  for  the  DoD 

Energy  Saving  Contracts/DoD  Mobile  Assets 

Financing  DoD  Budget  via  PPPs 

Lessons  from  Private  Sector  Capital  Budgeting  for  DoD  Acquisition  Budgeting 
Reform 

PPPs  and  Government  Financing 
ROI  of  Information  Warfare  Systems 
Special  Termination  Liability  in  MDAPs 
Strategic  Sourcing 

Transaction  Cost  Economics  (TCE)  to  Improve  Cost  Estimates 


Human  Resources 


Indefinite  Reenlistment 

Individual  Augmentation 

Learning  Management  Systems 

Moral  Conduct  Waivers  and  First-tern  Attrition 

Retention 

The  Navy’s  Selective  Reenlistment  Bonus  (SRB)  Management  System 
Tuition  Assistance 


Logistics  Management 


Analysis  of  LAV  Depot  Maintenance 
Army  LOG  MOD 
ASDS  Product  Support  Analysis 
Cold-chain  Logistics 

Contractors  Supporting  Military  Operations 
Diffusion/Variability  on  Vendor  Performance  Evaluation 
Evolutionary  Acquisition 

Lean  Six  Sigma  to  Reduce  Costs  and  Improve  Readiness 
Naval  Aviation  Maintenance  and  Process  Improvement  (2) 

Optimizing  CIWS  Lifecycle  Support  (LCS) 

Outsourcing  the  Pearl  Harbor  MK-48  Intermediate  Maintenance  Activity 
Pallet  Management  System 
PBL  (4) 

Privatization-NOSL/NAWCI 
RFID  (6) 
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■  Risk  Analysis  for  Performance-based  Logistics 

■  R-TOC  AEGIS  Microwave  Power  Tubes 

■  Sense-and-Respond  Logistics  Network 

■  Strategic  Sourcing 

Program  Management 

■  Building  Collaborative  Capacity 

■  Business  Process  Reengineering  (BPR)  for  LCS  Mission  Module  Acquisition 

■  Collaborative  IT  Tools  Leveraging  Competence 

■  Contractor  vs.  Organic  Support 

■  Knowledge,  Responsibilities  and  Decision  Rights  in  MDAPs 

■  KVA  Applied  to  AEGIS  and  SSDS 

■  Managing  the  Service  Supply  Chain 

■  Measuring  Uncertainty  in  Earned  Value 

■  Organizational  Modeling  and  Simulation 

■  Public-Private  Partnership 

■  Terminating  Your  Own  Program 

■  Utilizing  Collaborative  and  Three-dimensional  Imaging  Technology 

A  complete  listing  and  electronic  copies  of  published  research  are  available  on  our  website: 
www.acquisitionresearch.org 
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Defense  Acquisition  in  Transition 
6th  Annual  Symposium 


Acquisition  Risks  in  a  World  of  Joint  Capabilities 


Mary  Maureen  Brown 

University  of  North  Carolina  at  Charlotte 

Anita  Raja 

University  of  North  Carolina  at  Charlotte 

Robert  Flowe 
AV  SOA  AT&L 


p 

Program  Networks 


This  material  is  based  upon  work  supported  by  the  Naval  Postgraduate  School  Acquisition  Research 
Program  under  Grant  No.  N00244-10-1-0019 
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Joint  Capabilities 


Join  Capabilities  and  Network  Centric  Warfare 

is  an  emerging  theory  of  war  based  on  the 
concepts  of  nonlinearity,  complexity,  and  chaos.  It 
is  less  deterministic  and  more  emergent;  it  has  less 
focus  on  the  physical  than  the  behavioral; 

and  it  has  less  focus  on  things  than  on  relationships 


ADM  Cebrowski 


SCI  1001 


Complexity  and  Joint  Capabilities 


Nonlinear  interaction 


Decentralized  Control 


Self-Organization 
Non-equilibrium  Order 


Adaptation 


Combat  forces  composed  of  a  large  number  of 
nonlinearly  interacting  parts 

There  is  no  master  “oracle”  dictating  the  actions  of 
each  and  every  combatant 

Local  action,  which  often  appears  “chaotic,”  induces 
long-range  order 

Military  conflicts,  by  their  nature,  proceed  far  from 
equilibrium.  Correlation  of  local  effects  is  key 

Combat  forces  must  continually  adapt  and  coevolve  i 
a  changing  environment 


Collectivist  Dynamics 


There  is  a  continual  feedback  between  the  behavior 
combatants  and  the  command  structure 


Weapon  Systems  Acquisition 
Reform  Act  (WSARA)  of  2009 
Enacted  as  Public  Law  1 11-23 
on  May  22,  2009 


Implementing  Management  for 
Performance  and  Related  Reforms 
to  Obtain  Value  in  Every 
Acquisition  Act,  or  the 
IMPROVE  Acquisition  Act,  by  417- 
3  on  April  28. 


Acquisition  Reforms 


Challenges  with  the  requirements  process 
are  a  major  factor  in  poor  acquisition 
outcomes 

The  requirements  process  for  the  acquisition 
of  services  is  almost  entirely  ad  hoc. 

The  process  for  developing  requirements  for 
the  acquisition  of  weapon  systems  lacks  the 
expertise  and  capacity  required  to  vet  joint 
military  requirements. 

Joint  staff  lacks  some  of  the  analytical 
expertise  necessary  to  ensure  that  the 
JCIDS  process  rigorously  vets  proposed 
requirements 


Joint  Capabilities 


An  integrated  approach  to  strategic  planning,  capabilities  needs 
assessment,  systems  acquisition,  and  program  and  budget  development. 


To  identify  and  assess 
joint  military  capability 
needs  that  serve  as  the 
basis  for  the  development 
and  production  of 
acquisition  programs 


To  provide  joint  analytic  decision 
support  with  PPBE  milestones 


F  joirvT  kapobiUfftes- 
Integration  and 

Jjc»i 


Planning, 
Programming, 
Budgall  ng . 
and  Ex&cultoii 
fnnnrj 
Process  v 


Bolen 30 
Acq  u  is  i  tion 
Syste  m 


To  assess  and  resolve 
gaps  in  military  joint 
warfighting  capabilities.  To 
effectively  integrate 
capabilities  identification 
and  acquisition  provide 
capabilities-based 
approach  to  requirements 
generation 


Because  the  future  operating  environment  will  be  characterized  by 
uncertainty,  complexity,  rapid  change,  and  persistent  conflict,  DoD 
leadership  has  explicitly  sought  the  capability  to  act  jointly 
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Interdependency  ::  Complexity 


Complexity  is  based  on 
relations,  and  by 
extension, 


principles  of  organization 
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Research  Goals 


Are  to  identify  the 


Characteristics, 
Behavior,  and 
•  Effects 


of  the  Programmatic  Networks  that  drive  Joint 
Capabilities  and  Network  Centric  Activities 


•  Incomplete  Information 

•  Incomplete  Payoff  Structures 

•  Inability  to  Isolate  Cause  and  Effect 

•  Unknown  Response  Options 

•  Multiple  and  Conflicting 
Representations  of  Environmental  Variety 

•  Perturbations 


Multiple  Constraints 


labor  allocations 

•  production 
•  consumption 

•  investment 

decisions 


Bandwidth 


Congestion 


Noise 


Stability 


Redundancy 


f ransaction  Costs 


Reliability 


Integrity 


Cost  Overruns 


Schedule  Delays 


Performance 


Feature  Shortfalls 
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Interdependency  Dimensions 


Direction 

Pooled 

Sequential 

Resource 

Financial 

Reciprocal 

Data 

Agency 

Authority 

Labor 

Service 

Information 

Government 

International 

Contractor 

Pattern  Illustrations 
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Pattern  &  Binding  Illustrations 


Program,  Link,  Tier,  Pattern  Knowledge 
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Knowledge  of: 

Bandwidth 

Congestion 

Noise 

Redundancy 

Instability 


Pgm  Link 


Value  Proposition 


Pgm  Link 


Value  Proposition 
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Research  Objectives 


Applied  Research  ::  2010 


•  Map  program  interdependence  to  reveal  the  directionality  of  influence  of 
cause-effect  relationships 

•  Test  the  cascading  risks  that  upstream  programs  exert  on  downstream 
programs  in  light  of  data  and  funding  exchanges 

•  Test  the  extent  to  which  the  cost  overruns  &  schedule  delays  of  upstream 
programs  cascade  on  to  interdependent  downstream  programs 

•  Employ  the  findings  to  make  recommendations  on  potential  governance 
mechanisms  that  may  prove  capable  of  mitigating  the  risks  of 
interdependencies 

•  Provide  a  research  code  book  of  acquisition  data  elements  for  future 
research  efforts 
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Data  Interdependencies 


Growing  Interdependencies  and  Growing  Complexity 


NAVAL 

raiGRADlMl  E 
SCKOOI 


Program  Element  Interdependencies 


Growing  Interdependencies  and  Growing  Complexity 
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Program  Element  Interdependencies 


Growing  Interdependencies  and  Growing  Complexity 


2005 
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Information  Value 


Nonlinear  and  Linear  Methods 
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Datasets 


AVSOA 
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Variables 


Data  Exchange 

APB  Schedule  :: 

APB  Performance  Breaches  :: 
APB  RDT&E  Breaches  :: 

PAUC  Breaches 

Interdependencies 

Total  Cost  Variance  :: 
Engineering  Cost  Variance  :: 
Schedule  Cost  Variance  :: 
Estimation  Cost  Variance  :: 

Percent  Cost  Growth 

NAVAL 
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eliminary  Results:  Correlation  Coefficients 


Program  Manager  s  Perception  of  Data  Risk  (2005-2007) 


Engineering  Cost  Variance 

-.08* 

Performance  Breaches 

.11*  Self 

RDT&E  Breaches 

.13** 

Total  Cost  Variance 

-.12** 

Engineering  Cost  Variance 

-.22*" 

Downstream 

Estimation  Cost  Variance 

-.11* 

Performance  Breaches 

.13** 

- - - 

RDT&E  Breaches 

.09* 

PAUC  Breaches 

.12** 
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Correlation  Coefficients 


Preliminary  Results  ::  Lagged  by  One  Year 

Sender  APB  Performance  Breaches  :: 

Downstream  RDT&E  Breaches  .07* 

Upstream 

Sender  Total  Cost  Variance  :: 

Influence  on 

Downstream  Schedule  Cost  Variance  .09* 

Downstream 

Programs 

Sender  Engineering  Cost  Variance  :: 

Downstream  Percent  Cost  Growth  .  1 2** 
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Network  of  10  nodes  in  2006 
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Network  Evolution  over  Time 


From  the  perspective  of  PNO  180 


NAVAL 


POSTGAADUAl  E 
SCKOOI 


Markov  Decision  Processes 


Ny  is  key. 


Interested  in  studying  the  effects  of 
subtle  changes  on  the  overall 
behavior  of  the  MDAP  network  ! 


nes  over  time  to  support 


MDi- . 


Computing  optimal  p 
support  non-myopic  o 


( 


MDP 


) 


First  order  and  second 
interdependencies 
among  MDAPS 


^quential  decision  making, 
gramming  will 


And  sequential 
effects! 
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Sample  MDP  of  a  MDAP  network 


Legend: 

MDAP  n/w 
state 


Reward: 

RedrBreach 

Green  :  No  breach 
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Distributed  Constraint  Optimization 


Study  effect  of  distributed  “What  if  questions  on  MDAP  n/w 
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Blocks 


Missing  Data  in  a  10  node  network 


•  PAUC  data  from  2002  to  2006  is  incomplete: 

For  e.g.  Data  for  critical  node  PNO  374  is  missing. 

•  Funding  proportion  data  from  2004-2007  is  incomplete: 

PNO  180  only  has  2005-2007  data. 

•  ~  40%  of  "PNO  spending  under  PE"  data  in  this  set  not 
available. 


•  Map  program  interdependence  to  reveal  the 
directionality  of  the  influence  of  cause-effect 
relationships 

Add  2008  -  2009 

•  Test  the  cascading  risks  that  upstream  programs 
exert  on  downstream  programs  in  light  of  data  and 
funding  exchanges 

Test  2005  -  2009 

•  Test  the  extent  to  which  the  cost  overruns  & 
schedule  delays  of  upstream  programs  cascade  on 
to  interdependent  downstream  programs 

Test  2005  -  2009 

•  Employ  the  findings  to  make  recommendations  on 
potential  governance  mechanisms  that  may  prove 
capable  of  mitigating  the  risks  of  interdependencies 

December 

•  Provide  a  research  code  book  of  acquisition  data 
elements  for  future  research  efforts 

December 

Back  up  Slides 


Markov  Decision  Process  (MDP)  model 


MDP  Factored  State  Features: 


FO:  Year 

FI:  Current  PNO  ID  and  %  change  in  its  PAUC 
F2:  Set  of  PE(s)  funding  PNO 
F3:  Engineering  cost  variance 
F4:  Schedule  cost  variance 
F5:  Estimation  cost  variance 

F6:  PEs  with  funding  relationships  and  PAUC  %  change 
F7:  PEs  with  data  relationships  and  PAUC  %  change 
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Action,  Transition,  Reward 


MDP  model 


Action  space:  Cross  product  of  diversity  features 


<Total  #  of  PES>  X  <#  of  funding  services> 


Other  diversity  features  being  studied  are  level  of  funding; 
command  levels;  #  of  inti  partners;  joint  requirements. 


Transition  Probabilities:  Obtained  statistically  from  generalizations  of 
past  data  from  2002-2007 


Reward  Function:  Based  on  Nunn  -  Mccurdy  breach  threshold 


Red:  PAUC%  >15%; 
Yellow:  5% -15% 
Green:  PAUC  %<  5% 
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Reward  transition  for  PNO  180 
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Distributed  Constraint  Optimization 


Given: 

•  Variables  {xl,  x2,  xn}, 

•  Finite,  discrete  domains  D1,  D2,  ,  Dn, 

•  For  each  xi,  xj,  valued  constraint  fij:  Di  x  Dj  -»  N. 

Goal: 

•  Find  complete  assignment  A  that 
maximizes/minimizes  F(A)  where,  F(A)  =  S  fjM.d), 


Xi^dj.Xj  -^dj  in  A 


< 


5 

MDP 
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^alue  of  Information  in  Decision  Networks 


Supporting  Joint  decision  making  by  multiple  Program 

MananarQ 


Value  of  Computation 

•  Captures  the  value  of  being  able  to  know  "not  only  additional 
uncertainties  but  also  additional  decisions  already  made  by  other 
team  members"  before  making  some  other  decisions  in  the  team 
decision  situation. 

Influence  diagram 

•  Generalization  of  a  Bayesian  network 

•  Structured  to  accommodate  team  decision  situation  where 
incomplete  sharing  of  information  among  team  members  can  be 
represented  and  solved  very  efficiently. 


